Skip to content

Stable 26.8 Backport of #128217 - Evaluate the PARTITION BY expression of file and object storage writes once per INSERT, not once per block - #2911

Merged
zvonand merged 2 commits into
stable-26.8from
backports/stable-26.8/128217
Nov 20, 2026
Merged

zvonand merged 2 commits into
stable-26.8from
backports/stable-26.8/128217

Conversation

@blau-ai

@blau-ai blau-ai commented Nov 18, 2026

Copy link
Copy Markdown
Collaborator

Changelog category (leave one):

  • Bug Fix (user-visible misbehavior in an official stable release)

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Fixed PARTITION BY of file, url and object storage writes (table functions and table engines) evaluating stateful and non-deterministic functions anew for every block since 26.1: PARTITION BY intDiv(rowNumberInAllBlocks(), N) restarted its counter on every block and wrote rows into the wrong files, and now64() produced one file per block instead of one per INSERT. Closes ClickHouse#128211. (ClickHouse#128217 by @groeneai)

Documentation entry for user-facing changes

...

CI/CD Options

Exclude tests:

  • Fast test
  • Integration Tests
  • Stateless tests
  • Stateful tests
  • Unit tests
  • Performance tests
  • Aarch64 tests
  • All with ASAN
  • All with TSAN
  • All with MSAN
  • All with UBSAN
  • All with Coverage
  • All Regression
  • Disable CI Cache

Regression jobs to run:

  • Fast suites (mostly <1h)
  • Aggregate Functions (2h)
  • Alter (1.5h)
  • Benchmark (30m)
  • CAS (content-addressed storage; Antalya only)
  • ClickHouse Keeper (1h)
  • Iceberg (2h)
  • LDAP (1h)
  • OAuth (5m)
  • Parquet (1.5h)
  • RBAC (1.5h)
  • SSL Server (1h)
  • S3 (2h)
  • S3 Export (2h)
  • Swarms (30m)
  • Tiered Storage (2h)

…tateful-function-spans-chunks

Evaluate the `PARTITION BY` expression of file and object storage writes once per INSERT, not once per block
@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

Workflow [PR], commit [05bef70]

@zvonand

zvonand commented Nov 20, 2026

Copy link
Copy Markdown
Member

@blau-ai

@blau-ai

blau-ai commented Nov 20, 2026

Copy link
Copy Markdown
Collaborator Author

CI triage for 05bef70

Summary: 4 failing checks. None of them come from this PR. 1 is an infra timeout, 1 is a fuzzer crash in an unrelated code path, 1 is an unrelated stateless test timeout, and 1 is the DCO check that fails on every backport PR. The new test 05459_partition_by_stateful_function_spans_chunks passed all 5 of its runs (0.35–0.42 s).

Check Failure Classification
Stateless tests (amd_debug, parallel) 03927_autopr_input_bytes_estimation_prewhere_filter timeout Not PR-related (flaky/timeout)
BuzzHouse (amd_debug) Inconsistent AST formatting on RESTORE ... SETTINGS Not PR-related (fuzzer bug in another area)
SQLLogic test Job hit its 3 h timeout Infra / slow runner
DCO Merge commits have no Signed-off-by Same on all backport PRs

1. Stateless tests (amd_debug, parallel): 03927_autopr_input_bytes_estimation_prewhere_filter

[13592 / 13592] 03927_autopr_input_bytes_estimation_prewhere_filter:  [ FAIL ] 1200.04 sec.
Reason: Timeout! Killing process group 667108
  • This is a stateful, long test that only runs SELECT ... FROM test.hits with parallel replicas and then reads query_log. It doesn't write files or object storage, and it doesn't use PARTITION BY, so PartitionedSink/IPartitionStrategy never run.
  • The client stack shows it waiting in ClientBase::receiveResult until the 1200 s limit.
  • The runner's diagnosis says the test passes without randomized settings (17/17). The culprit list includes things like max_threads 3, local_filesystem_read_method mmap, min_bytes_to_use_direct_io 1 and compile_expressions 1, which makes a slow debug build slower.
  • On other stable-26.8 PRs the same job passes, e.g. Stable 26.8 Backport of #131001 - Reuse the GCM cipher context between rows in encrypt/decrypt #2902 (Failed: 0, Passed: 13468).
  • Next step: safe to re-run. If it keeps timing out on stable-26.8, the test should pin more settings (it already overrides max_threads for this reason). That would be a separate change.

2. BuzzHouse (amd_debug): server abort

<Fatal> : Logical error: 'Inconsistent AST formatting: the query:
RESTORE TABLE system.documentation FROM Disk('local_cache', 'backup15.tgz') SETTINGS ...
cannot parse query back from RESTORE TABLE `system`.`documentation` FROM Disk(...) SETTINGS iceberg_snapshot_id = DEFAULT, ...
  • The abort comes from the AST format/parse round-trip check in executeQuery.cpp:2438. Formatting a RESTORE query injects iceberg_snapshot_id = DEFAULT into SETTINGS, and the result doesn't parse back.
  • That is the backup/restore AST code, which this PR doesn't touch. The PR only changes how a PartitionedSink evaluates PARTITION BY at INSERT time.
  • Next step: not a blocker for this PR. It's worth its own issue against stable-26.8, covering RESTORE settings formatting with iceberg_snapshot_id.

3. SQLLogic test: job timeout

WARNING: Timeout exceeded [10800] for [2783]
WARNING: Job timed out: [SQLLogic test], timeout [10800], exit code [-15]
  • The Complete test stage started at 18:53 and was still running when the job was killed at 21:04. No test failure is reported.
  • On Stable 26.8 - Bump to v26.8.39 #2874 (stable-26.8 version bump) this job passed but took 9185 s, already close to the 10800 s limit. This run took 10904 s, so it's a slow runner on a job with little margin.
  • Next step: re-run.

4. DCO

PR health

The diff is a straight port of upstream ClickHouse#128217. The partition-key actions are now built once in PartitionedSink's constructor and reused for every chunk, so stateful functions such as rowNumberInAllBlocks() and now64() behave consistently within one INSERT. The new test passes, and no failure in this run touches the changed code. Recommendation: re-run the Stateless and SQLLogic jobs. The BuzzHouse crash can be tracked separately.

@zvonand
zvonand marked this pull request as ready for review November 20, 2026 13:29
@zvonand
zvonand merged commit 4e9592a into stable-26.8 Nov 20, 2026
281 of 289 checks passed
@zvonand zvonand added verified Approved for release stable-26.8 labels Nov 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants